Systems and methods for shifting haptic feedback function between passive and active modes

ABSTRACT

Systems and methods for shifting haptic feedback function between passive and active modes are disclosed. For example, one disclosed method includes receiving a first signal from a sensor, the first signal associated with a mode of interaction with a graphical user interface; receiving a second signal associated with an interaction with the graphical user interface; determining a haptic feedback effect based at least in part on the mode of interaction with the graphical user interface and the interaction with the graphical user interface; and generating a haptic signal configured to output the haptic feedback effect.

CROSS-REFERENCES TO RELATED APPLICATIONS

This patent application claims the benefit of and is a continuation ofU.S. patent application Ser. No. 15/443,482, filed Feb. 27, 2017, andentitled “Systems and Methods For Shifting Haptic Feedback FunctionBetween Passive and Active Modes,” which claims the benefit of and is acontinuation of U.S. patent application Ser. No. 14/712,358, filed onMay 14, 2015, and entitled “Systems and Methods for Shifting HapticFeedback Function Between Passive and Active Modes,” which claims thebenefit of and is a continuation of, U.S. patent application Ser. No.13/832,420, filed on Mar. 15, 2013, and entitled “Systems and Methodsfor Shifting Haptic Feedback Function Between Passive and Active Modes,”which claims the benefit of and is a continuation of U.S. patentapplication Ser. No. 12/502,758, filed on Jul. 14, 2009, and entitled“Systems and Methods for Shifting Haptic Feedback Function BetweenPassive and Active Modes,” which claims the benefit of: U.S. ProvisionalPatent Application No. 61/080,978, entitled “Systems and Methods forPhysics-Based Tactile Messaging” filed Jul. 15, 2008; U.S. ProvisionalPatent Application No. 61/080,981, entitled “Systems and Methods forMapping Message Contents to Virtual Physical Properties for VibrotactileMessaging” filed Jul. 15, 2008; U.S. Provisional Patent Application No.61/080,985, entitled “Systems and Methods for Shifting Sensor HapticFeedback Function Between Passive and Active Modes” filed Jul. 15, 2008;U.S. Provisional Patent Application No. 61/080,987, entitled “Systemsand Methods for Gesture Indication of Message Recipients” filed Jul. 15,2008; U.S. Provisional Patent Application No. 61/148,312, entitled“Systems and Methods for Pseudo-Telepresence in a Shared Space” filedJan. 29, 2009; and U.S. Provisional Patent Application No. 61/181,280,entitled “Systems and Methods for Transmitting Haptic Messages” filedMay 26, 2009, the entirety of each of which is hereby incorporated byreference herein.

FIELD OF THE INVENTION

The present invention generally relates to haptic feedback and moreparticularly to systems and methods for shifting haptic feedbackfunction between passive and active modes.

BACKGROUND

Over the past several years, the use of handheld devices of all typeshas grown exponentially. These devices are used as portable organizers,telephones, music players, and gaming systems. Many modern handhelddevices now incorporate some type of haptic feedback. As haptictechnology improves, devices may incorporate multiple modes of hapticfeedback. A way to switch between and select modes of feedback isneeded.

SUMMARY

Embodiments of the present invention provide systems and methods forshifting between active and passive modes of haptic feedback. Forexample, in one embodiment, a method for shifting between active andpassive modes of haptic feedback comprises receiving a first signal froma sensor, the first signal associated with a mode of interaction with agraphical user interface; receiving a second signal associated with aninteraction with the graphical user interface; determining a hapticfeedback effect based at least in part on the mode of interaction withthe graphical user interface and the interaction with the graphical userinterface; and generating a haptic signal configured to output thehaptic feedback effect. In another embodiment, a computer-readablemedium comprises program code for carrying out such a method.

These illustrative embodiments are mentioned not to limit or define theinvention, but rather to provide examples to aid understanding thereof.Illustrative embodiments are discussed in the Detailed Description,which provides further description of the invention. Advantages offeredby various embodiments of this invention may be further understood byexamining this specification.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages of the presentinvention are better understood when the following Detailed Descriptionis read with reference to the accompanying drawings, wherein:

FIG. 1 is a block diagram of a system for shifting haptic feedbackfunction between passive and active modes according to one embodiment ofthe present invention;

FIG. 2 is a flow diagram of a method for mapping message contents tovirtual physical properties for vibrotactile messaging according to oneembodiment of the present invention;

FIG. 3 is another illustration of shifting haptic feedback functionbetween passive and active modes according to one embodiment of thepresent invention;

FIG. 4 is another illustration of shifting haptic feedback functionbetween passive and active modes according to one embodiment of thepresent invention;

FIG. 5 is yet another illustration of shifting haptic feedback functionbetween passive and active modes according to one embodiment of thepresent invention;

FIG. 6 is a final illustration of shifting haptic feedback functionbetween passive and active modes according to one embodiment of thepresent invention; and

FIG. 7 is an illustration of a virtual object that may be displayedaccording to one embodiment of the present invention.

DETAILED DESCRIPTION

Embodiments of the present invention provide systems and methods forshifting haptic feedback function between passive and active modes.

Illustration of Shifting Haptic Feedback Function Between Passive andActive Modes

One illustrative embodiment of the present invention comprises amessaging device, such as a mobile phone. In the illustrativeembodiment, the messaging device comprises the Samsung SGH-i710 mobilecomputer equipped with Immersion Corporation's VibeTonz® vibrotactilefeedback system. In another embodiment, the messaging device comprisesImmersion Corporations TouchSense® Technology system also known asImmersion TouchSense® vibrotactile feedback system. Other messagingdevices and haptic feedback systems may be utilized.

The messaging device comprises a display, a user interface device,memory, and a processor in communication with each of these elements.The illustrative messaging device also comprises a sensor and anactuator, both of which are in communication with the processor. Thesensor is configured to sense a user interaction with the messagingdevice and the actuator is configured to output a haptic effect.

In the illustrative device, the processor receives a first signalassociated with a mode of interaction with a messaging device'sgraphical user interface. The processor may receive the first signalfrom a sensor, such as a gyroscope or an accelerometer in the housing ofthe messaging device. Such a sensor may be configured to detect when auser moves or tilts the messaging device. A particular movement of thedevice may then be associated with a particular mode of interaction,such as a passive or active mode. For example, to engage an active modeof interaction, the messaging device may be tipped or rotated in apredetermined direction; to switch to a passive mode of interaction, themessaging device may be tipped or rotated in the opposite direction.

Next, the processor receives a second signal from the user interfacedevice associated with an interaction with the graphical user interface.For instance, the processor may receive the second signal from atouch-sensitive input device, such as a touch-screen. The interactionmay take a number of forms. For example, a user may brush, flick, rub,drag, or otherwise engage the touch-screen display to interact with avirtual object displayed on the graphical user interface. Alternatively,after a mode of interaction has been selected, a user may tilt, shake,rotate, or move the messaging device to interact with the graphical userinterface. In an active mode of interaction, brushing or flicking avirtual object in the graphical user interface may cause the virtualobject to move within the graphical user interface, while the processorgenerates a haptic signal simulating the virtual object's weight. In apassive mode of interaction, touching a virtual object, may cause theprocessor to generate a haptic signal simulating the virtual object'ssurface features, such as its texture. The simulation of the surface maytake the form of a vibration or other haptic feedback effect.

After receiving both signals, the processor determines a haptic feedbackeffect based, at least in part, on the mode of interaction with thegraphical user interface and the interaction with the graphical userinterface. In an illustrative embodiment, a first signal is associatedwith an active mode of interaction and the second signal is associatedwith a user interaction with an object displayed on the graphical userinterface. The user's interaction may involve moving a virtual object onthe graphical user interface. The processor may then determine hapticfeedback, which resembles a collision between the virtual object andanother surface in the graphical user interface. Alternatively, if thefirst signal is associated with a passive mode of interaction and thesecond signal is associated with a rubbing motion on or near a virtualobject, the processor may determine haptic feedback simulating thetexture of the virtual object. Finally, the processor generates a hapticsignal configured to cause an actuator to output the haptic effect. Anactuator receives the haptic signal, and generates the haptic effect,for example by shaking, jolting, or vibrating the messaging device.

This illustrative example is given to introduce the reader to thegeneral subject matter discussed herein. The invention is not limited tothis example. The following sections describe various additionalnon-limiting embodiments and examples of methods and systems forshifting haptic feedback function between active and passive modes.

Shifting Haptic Feedback Function Between Passive and Active Modes

Embodiments of the invention presented herein provide systems andmethods for shifting haptic feedback function between passive and activemodes. A modern handheld device may include the capability to includemultiple modes of feedback. These modes may be movement based “activemodes” or texture based “passive modes.” The modes of feedback may beoutput to the user visually through the display, audibly using speakers,or haptically using actuators. The claimed invention provides systemsand methods for switching between active and passive modes of feedback.

For example, in a messaging device including an accelerometer and atouch-screen, a default mode of interaction may comprise an active modeof interaction. In such a mode, tilting the messaging device may causevirtual objects displayed on a graphical user interface to appear toroll around. In response to the movement of the virtual objects, thedevice may generate haptic feedback representing friction and impactsbetween the virtual objects and other surfaces. Using such a device, auser may also utilize the touch-screen to directly move a virtual objectwithin the user interface. The user may tilt the messaging device tocause the virtual object to move on the graphical user interface. Or theuser may directly manipulate the virtual object by touching thegraphical user interface. The movement of the virtual object on thegraphical user interface causes an actuator to generate one or morehaptic effects based on the virtual object's friction profile.

Each virtual object comprises a virtual physical parameter which definesone or more of the object's size, mass, shape, collision behavior, andtexture. The virtual physical parameter may also include the object'stexture and friction profiles. The texture profile is a data storecomprising parameters that define the object's reaction to userinteraction in the passive mode of interaction. The texture profile mayfurther define the virtual object's surface features. The frictionprofile is a data store comprising parameters defining the virtualobject's reaction to user manipulations and its interaction with othervirtual objects in the active mode of interaction. The friction profilemay additionally define other properties, such as virtual height,weight, density, etc. When the user interacts with the virtual object orthe virtual object interacts with other features on the graphical userinterface, its movement and the haptic feedback in response to theinteraction will be based, at least in part, on the virtual object'sfriction profile. For example, a ball could have a friction profilewhich causes the ball to respond to interaction by bouncing whileanother ball may respond by rolling. Furthermore, a dense heavy ball maymove slowly with hard impacts on surfaces within the interface; while alighter, less dense, ball may move quickly with softer impacts. Avirtual object may be one of any number of shapes, including, but notlimited to: a ball, an egg, a tube, a capsule or a box.

A virtual object may represent a type of data. For example, a virtualobject may represent a video file within the device. Additionally, theshape chosen for a virtual objects shape may take into account the typeof data represented by a virtual object. For example, a virtual objectrepresenting a picture may take the shape of a camera, while a virtualobject representing a video file may take the for of a compact disk.

A virtual object may comprise a token A token is a type of virtualobject that includes predefined parameters. The user may then defineadditional parameters of a specific instantiation of a token. Forexample, in one embodiment a token may comprise virtual object in theform of a ball, with predefined friction and texture profiles. The usermay then define additional characteristics, such as the ball's size andcolor.

A processor may receive a shift or change signal associated with adifferent mode of interaction and switch to a passive mode ofinteraction. In the passive mode of interaction, haptic effects may bebased on a texture profile of a virtual object rather than the frictionprofile. A texture profile is a data store that contains datarepresenting surface features of a virtual object, such as texture andtension. When a user contacts a virtual object, the device may generatehaptic effects that simulate the surface of the object. For example, avirtual object could have a rough surface indicated by rapid hard hapticpulses, while a virtual object with a soft surface could be indicated bya steady gentle vibration.

In some embodiments, the user interface may be represented as a virtualmessage environment programmed to incorporate a physical model. In suchan environment, electronic messages may be displayed as a virtualmessage objects. A virtual message object is a form of virtual objectthat includes a data store incorporating an electronic message. Avirtual message object comprises a virtual physical parameter whichdefines one or more of the object's size, mass, shape, collisionbehavior, and texture. The virtual physical parameter may also includethe object's texture and friction profiles. The texture profile is adata store comprising parameters that define the object's reaction touser interaction in the passive mode of interaction. The texture profilemay further define the virtual object's surface features. The frictionprofile is a data store comprising parameters defining the virtualobject's reaction to user manipulations and its interaction with othervirtual objects in the active mode of interaction. The friction profilemay additionally define other properties, such as virtual height,weight, density, etc. A user can manipulate the virtual messageenvironment and virtual message objects through various messaging deviceinterfaces and sensors, such as a touch-screen, gyroscope, GPS, oraccelerometer, or other sensor configured to detect movement.

A virtual message object may comprise a token A token is a type ofvirtual object that includes predefined parameters. The user may thendefine additional parameters of a specific instantiation of a token. Forexample, in one embodiment a token may comprise virtual object in theform of a ball, with predefined friction and texture profiles. The usermay then define the text contained by the instantiation of the token aswell as additional characteristics, such as the ball's size and color.

When a user is interacting with the device, a sensor detects the user'smovements and transmit signals representing these movements to theprocessor. The processor may then calculate vectors which represent thedisplacement and magnitude of the users movement. The processor may thentranslate these vectors into virtual forces acting on virtual messageobjects contained within the virtual message environment. If the messageis sent to a haptically enabled device, the haptic effect may beincorporated as a component of the message. For example, a user mayincorporate a soft gesture into a message. When the recipient receivesthe message, this soft gesture may be translated into a soft hapticeffect, indicating that a message is not urgent. Conversely, a user mayincorporate a hard gesture into a message, which will be translated intohard haptic force indicating an urgent message. Like other virtualobjects, virtual message objects may have both a texture profile and afriction profile, which may be used to calculate haptic effects.

In one embodiment, when a user wishes to switch between modes offeedback, a gesture, position, key, or other control is assigned as aselector for the mode of interaction. For example, a user can use aspecific gesture to switch between active and passive modes ofinteraction, or to switch to another mode of interaction. The userinterface may indicate/illustrate the change in mode through the visualdisplay, a sound effect, and/or haptic feedback. For example, if theuser switches from passive to active modes, the device may give a visualindication through the display while also providing a haptic indication,such as a vibration.

In another embodiment, the device may be equipped with a switch orbutton, which may be displayed on a touch screen to switch between modesof interaction. In this embodiment, for example, a user may depress ashift key, indicating an active mode of interaction. While operating inthis mode, movements of the device can cause virtual objects in thedisplay to move, roll, and collide with each other. When the userreleases the shift key, the passive mode of interaction is enabled. Inthis mode, the virtual objects may exhibit different variables, such asa vibrating feedback representing each virtual object's texture. Whilein the passive mode, the virtual objects may continue movement on thescreen and the user may interact with the virtual objects; however, theuser cannot direct interaction between the virtual objects.

Illustrated System for Shifting Haptic Feedback Function Between Passiveand Active Modes of Interaction

Referring now to the drawings in which like numerals indicate likeelements throughout the several figures, FIG. 1 is a block diagram of asystem for shifting haptic feedback function between passive and activemodes according to one embodiment of the present invention. As shown inFIG. 1, the system 100 comprises a messaging device 102, such as amobile phone, portable digital assistant (PDA), portable media player,or portable gaming device. The messaging device 102 comprises aprocessor 110 in communication with a network interface 112, a sensor114, a display 116, an actuator 118, and a speaker 120. The processor110 is configured to generate a virtual environment, which is shown ondisplay 116.

The processor 110 is in communication with the network interface 112.The network interface 112 may comprise one or more methods of mobilecommunication, such as infrared, radio, Wi-Fi, or cellular networkcommunication. In other variations, the network interface 112 comprisesa wired network interface, such as Ethernet. The messaging device 102can be configured to exchange messages or virtual message objects withother devices (not shown) over networks such as a cellular phone networkand/or the Internet. Embodiments of messages exchanged between devicesmay comprise voice messages, text messages, data messages, or otherforms of digital messages.

The processor 110 is also in communication with one or more sensors 114.The sensor 114 may comprise a location sensor, rotational velocitysensor, image sensor, pressure sensor, or other type of sensor. Forexample, sensor 114 may comprise an accelerometer, a gyroscope, a GPSsensor, or a touch-sensitive input device (e.g. touch screen,touch-pad). The one or more sensors 114 may be configured to detectchanges in, for example, acceleration, inclination, inertia, orlocation. For example, the messaging device 102 may comprise anaccelerometer configured to measure the acceleration of the messagingdevice 102. As another example, the messaging device 102 may comprise alocation sensor, rotary velocity sensor, image sensor, pressure sensor,or other type of sensor. The one or more sensors 114 may be configuredto send a sensor signal to the processor 110.

Users may interact with the user interface through movements or gestureswhich are detected by the one or more sensors 114. As the messagingdevice 102 is tilted, shaken, or otherwise moved, the one or moresensors 114 may detect these movements and generate sensor signals sentto the processor 110 based, at least in part, on the movements. Theprocessor may use these signals to generate vectors with a direction andmagnitude that represents the direction and magnitude of the user'sinteraction with the device. In one embodiment, an accelerometer sensoris configured to detect the inclination and acceleration of themessaging device 102. As the messaging device 102 is tilted, anaccelerometer can be configured to send signals to the processor 110based, at least in part, on the tilt and/or acceleration of themessaging device 102. The processor 110 then uses these signals togenerate vectors representing direction and magnitude of theinteraction.

Signals received from the sensor 114 may be associated with a mode ofinteraction with a graphical user interface. For example, a jabbingmotion detected by an accelerometer may be associated with an activemode of interaction, while a circular or rotational motion detected byan accelerometer may be associated with a passive mode of interaction.By mapping modes of interaction to particular gestures, the processor110 may be able to switch modes of interaction quickly and effectively.

Signals received from the sensor 114 may also be associated with aninteraction with the graphical user interface. For example, a user mayfirst select a mode of interaction by pressing or depressing a shiftkey. After the mode of interaction has been selected, the messagingdevice may be tilted, rotated, shaken, or otherwise moved to interactwith the graphical user interface. Each movement of the device detectedby a sensor 114 may be associated with an interaction with the graphicaluser interface. In one mode of interaction, tilting the device may zoomin or zoom out of the graphical user interface. For example, a digitalimage may be shown on the display 116. To manipulate the image, the usermay select an active mode of interaction, and then tilt the device suchthat sensor 114 detects the movement. The device may then manipulate theimage shown on display 116 based on the data received from sensor 114.The device could manipulate the image in many different ways; forinstance, the manipulation could involve zooming in or away from theimage, scrolling, or rotating the displayed image.

The processor 110 may receive signals associated with a mode ofinteraction and/or signals associated with an interaction from othersources (not shown in FIG. 1). For example, the messaging device 102 maycomprise a scroll wheel, a rocker switch, or a joystick (not shown inFIG. 1). The messaging device 102 may also comprise one or more buttons,such as a shift button (not shown in FIG. 1).

In the embodiment shown in FIG. 1, the processor 110 is also incommunication with a display 116. The processor 110 can be configured togenerate a graphical representation of a user interface to be shown ondisplay 116. The display 116 may comprise a touch-sensitive inputdevice, such as a touch screen, configured to send and receive signalsfrom the processor 110.

Signals received from a touch-screen display 116 may be associated witha mode of interaction with the graphical user interface. In onevariation, a predetermined gesture, such as a tap on a touch-screen, maybe associated with a specific mode of interaction. In one suchembodiment, one tap on the touch-screen may be associated with an activemode of interaction, while two taps on the touch-screen may beassociated with a passive mode of interaction. In another variation, agesture detected by a touch-screen in a particular area of the display116, such as an upper right quadrant, may be associated with one mode ofinteraction, while a gesture detected by a touch-screen in a differentarea of the display 116, such as a lower left quadrant, may beassociated with a second mode of interaction. Various other combinationsof gestures or user actions may be utilized to switch between modes ofinteraction.

In the embodiment shown in FIG. 1, the device can allow the user todirectly manipulate objects shown on display 116. If display 116comprises a touch-screen display, in an active mode of interactiontwo-dimensional finger gestures on display 116 may select, drag, flick,throw, or move a virtual object within the user interface. In a passivemode of interaction, two-dimensional finger gestures on display 116 maytouch and/or feel a virtual object without changing its location withinthe graphical user interface.

The processor 110 may determine a haptic feedback effect based, at leastin part, on the mode of interaction with the graphical user interfaceand the user's interaction with the graphical user interface. In theactive mode of interaction, the processor 110 may determine a hapticeffect simulating a jolt or a collision based on a user interaction,such as a jab or a flick. For example, the display may show a ball,which the user flicks. When the user flicks the image of the ball,processor 110 may determine a haptic effect representing a jolt to theuser. The jolt exhibiting characteristics of the virtual ball containedin its friction profile, such as the ball's weight, density, and type ofinteraction with other virtual objects. Processor 110 may furthercalculate a collision when the virtual ball impacts a side of display116. In a passive mode of interaction, processor 110 may determine ahaptic effect simulating a rough or smooth texture based on aninteraction such as a brush or rub. For example, in one embodiment,display 116 shows a ball with a texture profile comprising datarepresenting a rough surface. When the user then interacts with theball, processor 110 may calculate a haptic signal representing a seriesof haptic jolts, indicating the ball's rough surface.

As shown in FIG. 1, processor 110 is also in communication with one ormore actuators 118. Actuator 118 may be configured to receive a signalfrom processor 110 and generate a haptic effect. After processor 110determines a haptic effect, processor 110 may send a haptic signal tothe actuator 118 configured to cause actuator 118 to output the hapticeffect. Actuator 118 may be, for example, a piezoelectric actuator, anelectric motor, an electro-magnetic actuator, a voice coil, a linearresonant actuator, a shape memory alloy, an electro-active polymer, asolenoid, an eccentric rotating mass motor (ERM), or a linear resonantactuator (LRA).

Illustrative Method for Shifting Haptic Feedback Function BetweenPassive and Active Modes of Interaction

FIG. 2 is a flow diagram of a method for determining a haptic effectbased, at least in part, on a mode of interaction and a user'sinteraction with a graphical user interface. While embodiments of thecurrent invention may be utilized in a variety of devices, the processshown in FIG. 2 will be described in relation to the device shown inFIG. 1. In method 200, a processor 110 receives a first signalassociated with a mode of interaction with a graphical user interface202. In one embodiment, the signal indicates a change in the mode ofinteraction; in other embodiments it represents no change in the mode ofinteraction. A signal associated with a mode of interaction may begenerated by a sensor 114, such as an accelerometer, a gyroscope, a GPSsensor, a microphone, a touch-sensitive input device (e.g. touch screen,touch-pad), a texture stylus, an imaging sensor, or some other type ofsensor. Or the mode of interaction may be selected through a button orswitch. For example, a mode of interaction may be selected by rotatingthe messaging device 102 in a predetermined manner, depressing a shiftbutton, blowing into a microphone, or making a gesture on a touch-screendisplay 116. In one embodiment, the first signal is controlled in partby the processor. For example, the processor may receive an input, anduse this input to calculate and then transmit a signal which will changethe mode of interaction with the graphical user interface. In anotherembodiment, the first signal is controlled to return to a defaultsetting after a period of time. For example, the device may have adefault setting of the passive mode of interaction and, if the mode ofinteraction has been set to active for more than 5 minutes, theprocessor may switch the device back to passive mode.

The graphical user interface can display virtual objects with which auser may interact. In the embodiment shown, the graphical user interfacecomprises one or more modes of interaction. In one example, thegraphical user interface comprises an active mode of interaction and apassive mode of interaction. In an active mode of interaction, forexample, a user may direct interactions between one or more virtualobjects and other virtual objects and/or the virtual environment. In apassive mode of interaction, a user may interact only with a virtualobject without interacting with a virtual environment or causing thevirtual object to interact with other objects in the virtualenvironment.

After a mode of interaction is selected, the processor receives a secondsignal associated with an interaction with the graphical user interface204. The signal associated with an interaction may be generated by atouch-sensitive input device (e.g. touch screen, touch-pad) or someother type of sensor, such as an accelerometer, a gyroscope, a GPSsensor, a microphone, a texture stylus, or an imaging sensor. In oneexample, a user interacts with the user interface by contacting avirtual object through a touch-screen display 116. The processor 114receives a signal from the touch-screen display 116 associated with thegesture or contact made by the user. The signal associated with theinteraction may include properties of the interaction, such as location,speed, pressure, and/or type. For example, a signal from a touch-screendisplay 116 may include the location of the contact with thetouch-screen display 116, the speed and/or pressure of the contact, theduration of the contact, and the type of gesture or motion made on thetouch-screen display 116.

The first signal and the second signal may be received substantiallysimultaneously. For example, a user may depress a shift key and make agesture on a touch screen substantially simultaneously. In such ascenario, the processor receives the first signal associated with a modeof interaction at substantially the same time as processor 110 receivesa second signal associated with a user interaction with the userinterface. In another variation, processor 110 receives the secondsignal at some later period of time, such as a half second to severalseconds or more, after receiving the first signal. For example, a usermay select a mode of interaction by moving the messaging device in apredetermined manner. Later, a user may interact with the user interfaceby making a gesture on the touch-screen display 116 of the messagingdevice. The device may have a user-selected default mode of interaction.

The processor 110 determines a haptic effect based, at least in part, onthe mode of interaction with the graphical user interface and theinteraction with the graphical user interface 206. In a passive mode ofinteraction, haptic effects may be based on the texture profile of thegraphical user interface and/or a virtual object displayed in thegraphical user interface. In a passive mode of interaction, when a userrubs or brushes a virtual object, the processor may determine a hapticeffect simulating a texture of the virtual object, such as a vibrationsimulating a rough or smooth texture. In an active mode of interaction,haptic effects may be based on the friction profile of the graphicaluser interface and/or a virtual object displayed on the graphical userinterface. In an active mode of interaction, when a user rubs or brushesa virtual object, the virtual object may move on the display incorrelation to the user's movement. Then, as the virtual object moves inthe display, the processor may determine a haptic effect simulating themovement of the virtual object. For example, the processor may generatea long, low rumble for a heavy object, or a gentler vibration tosimulate a lighter object.

Next, the processor 110 generates a haptic signal configured to cause anactuator 118 to output the haptic effect 208. Finally, the processor 110transmits the haptic signal to one or more actuators 118 configured tooutput the haptic effect 210. After receiving the haptic signal, theactuator 118 then outputs the haptic effect. The haptic effect maycomprise many different types of feedback, including but not limited to:feedback representing texture, feedback representing tension, feedbackrepresenting grip, feedback representing a flick, feedback representinga stroke, feedback representing popping, feedback representing cracking,feedback representing movement, feedback representing friction, orfeedback representing a collision or collisions.

In one illustrative embodiment, the graphical user interface displays avirtual object in the shape of a heart. Then, processor 110 receives asignal associated with a passive mode of interaction. Next, processor110 receives a signal associated with a rub or a brush of the virtualheart. Then, based on both of the received signals and on the virtualheart's texture profile, processor 110 may determine a haptic effectsimulating a beating heart, which is output when the user interacts withthe virtual heart. Actuator 118 receives the haptic signal and generatesthe haptic effect simulating a beating heart.

In another illustrative embodiment, the graphical user interfacedisplays a virtual object in the shape of a heart. The processor 110first receives a signal associated with an active mode of interaction,and then receives a signal associated with a user's interaction with thevirtual heart. At this point, processor 110 may determine a hapticeffect based on both the received signals and the virtual heart'sfriction profile. The calculated effect may simulate the heart's weightas the user drags it across the screen. An actuator 118 receives ahaptic signal and generates the haptic effect simulating the heart'sweight.

In another illustrative embodiment, the graphical user interfacedisplays a virtual messaging environment. In the virtual messagingenvironment, an electronic message such as an email, text message,instant message, or other type of message is displayed as a virtualmessage object. In one embodiment, the virtual message object comprisesone of: a ball, a tube, a capsule, a box, a balloon, a heart, or anyother shape the graphical user interface is capable of displaying. FIG.7 shows a messaging device 710, that includes a display 720 showing avirtual message object in the form of a ball with a rough surface 730.The virtual message object further comprises a virtual physicalparameter which defines one or more characteristics of the virtualobject, such as: size, mass, shape, collision, or texture. A virtualphysical parameter may be modified by either the user or the programitself. In this illustrative embodiment, the processor 110 receives asignal associated with an interaction with the virtual message object inthe virtual messaging environment. Then the processor 110 receives asignal associated with an interaction with the virtual message object inthe virtual messaging environment. Processor 110 may then determine ahaptic effect based, at least in part, on the mode of interaction withthe virtual message object and the interaction with the virtualmessaging object.

For example, in one embodiment, the graphical user interface displays avirtual messaging environment and a virtual message object in the formof a balloon. The balloon comprises a virtual physical parameter whichdefines its size, shape, and collision behavior. The processor may thenreceive signal associated with an active mode of interaction, and mayfurther receive a signal associated with a user poking the balloon. Theprocessor will then calculate a haptic effect based, at least in part,on the mode of interaction and the interaction in this case, active modeof interaction and a poking interaction. The processor will thentransmit the calculated haptic signal to an actuator configured tooutput the haptic effect. The haptic effect may be, for example, asignal representing the virtual balloon popping. The processor mayfurther perform an action with the virtual message object, such assending the virtual message when the virtual balloon pops.

Illustrations of Shifting Haptic Feedback Function Between Passive andActive Modes of Interaction

FIG. 3 is an illustration of shifting haptic feedback function betweenpassive and active modes of interaction according to one embodiment ofthe present invention. In FIG. 3, a messaging device 302 comprises ashift button 304 and a touch-screen 306. The messaging device 302generates a graphical user interface, such as a virtual environment 310shown on the touch-screen 306.

As shown in FIG. 3, the graphical user interface 310 comprises onevirtual object 312 in the form of a ball. In other embodiments, thegraphical user interface may contain zero, one, or more virtual objects.When no objects are displayed, the background of the display may haveits own friction and texture profiles. When virtual objects aredisplayed, each virtual object comprises a virtual physical parameter.The virtual physical parameter may include one or more characteristicsof the virtual object, such as: size, shape, mass, collision behavior,or texture. The physical parameter may further include a friction and/ora texture profile which define the virtual object's interaction with theuser and with other virtual objects. For example, the friction profileof a virtual ball may comprise a rolling or bouncing profile. And thetexture feedback profile of a virtual ball may comprise a semi-rough ortacky surface. The processor may determine a haptic effect based, atleast in part, on a friction profile and/or a texture profile of avirtual object.

Each virtual object may comprise one or more virtual physicalparameters. A virtual physical parameter can comprise a size, a mass, ashape, a collision behavior, a texture, or a visual representation. Insome scenarios, a virtual object represents a message, such as a textmessage, a file message, or a voicemail. The virtual physical parametersof a virtual object may be based, at least in part, on the type of datathe virtual object represents. In one example, a virtual object mayrepresent a short text message and may comprise a light, thin,cylindrical object that is neither bouncy nor brittle. A virtual objectrepresenting a long video file may comprise a large, heavy oval objectthat cracks.

The friction profile and/or the texture profile may be based, at leastin part, on the virtual physical parameters of a virtual object. Forexample, the friction profile of an egg-like virtual message object maydictate a cracking or breaking action when the virtual egg moves in thevirtual environment. Thus, when a virtual egg object moves within avirtual environment and cracks or breaks, a haptic feedback effectsimulating the virtual egg cracking or breaking may be generated in onemode of interaction. In another mode of interaction, the egg may feelsmooth to the touch, represented as vibrations with a texturalcharacter.

A virtual object may comprise a token A token is a type of virtualobject that includes predefined parameters. The user may then defineadditional parameters of a specific instantiation of a token. Forexample, in one embodiment a token may comprise ball with predefinedfriction and texture profiles. The user may then define additionalcharacteristics of the ball, such as its size and color.

Before interacting with the user interface 310, a mode of interactionmay be selected. For example, the shift button 304 may be used to selecta mode of interaction with the user interface. As shown in FIG. 3, theshift button 304 is not depressed. When the shift button 304 is notdepressed, the user interface 310 may remain in a mode of interactionthat the user has defined as the default. When the shift button isactivated, or pressed, the user interface may switch to a different modeof interaction. While the shift button 304 is depressed, it may generatea signal indicating a change in mode of interaction. A processor (notshown in FIG. 3) of the messaging device 302 may receive the low signalfrom the shift button 304 and determine that the passive mode ofinteraction is intended. When a high signal is received from the shiftbutton 304, the processor may determine that an active mode ofinteraction is intended. In other variations, the default mode ofinteraction may comprise an active mode of interaction or some othermode of interaction. In other embodiments, the processor may control themode of interaction, and return the mode of interaction to auser-defined default after a user-defined period of time.

After the mode of interaction is selected, a user may interact with theuser interface 310 and/or the virtual object 312 by contacting thetouch-screen 306. In other variations, a user may interact with the userinterface 310 and/or the virtual object 312 by shaking, tilting,rotating, or otherwise moving the messaging device. For example, in oneembodiment, the display may show a virtual ball. In the active mode ofinteraction, the user may tilt the screen causing the virtual ball toroll inside the graphical user interface. The device may output a hapticeffect when the virtual ball rolls to a boundary of the graphical userinterface. In another embodiment, the virtual ball may represent a textmessage, email, or other electronic message. When the virtual ballrepresenting a message rolls to a boundary of the graphical userinterface, the device may allow the virtual ball to roll off thedisplay, thereby sending the message.

As shown in FIG. 3, the user interface 310 is in a first mode ofinteraction and a user is brushing or rubbing the touch-screen 306 inthe directions illustrated by arrow 320 and arrow 322. In a first modeof interaction, such as a passive mode, brushing or stroking the virtualobject 312 may cause the messaging device 302 to simulate or portray thetexture profile of the virtual object 312 through haptic feedback. Afterthe processor receives a signal associated with an interaction with thevirtual object 312, the processor determines a haptic effect simulatinga texture profile of the virtual object 312, such a vibration simulatingthe surface, composition, and/or material of the virtual object 312. Theprocessor then sends a haptic signal to an actuator (not shown in FIG.3) which generates the haptic effect.

FIG. 4 is an illustration of shifting haptic feedback function betweenpassive and active modes according to one embodiment of the presentinvention. In FIG. 4, a messaging device 402 comprises a shift button404 and a touch-screen 406. The messaging device 402 displays agraphical user interface, such as a virtual environment 410 shown on thetouch-screen 406. A virtual object 412, illustrated as a ball, is showninside the virtual environment 410.

As shown in FIG. 4, a user is depressing the shift key 404 in a downwardmotion indicated by arrow 424. At substantially the same time, orsubstantially simultaneously, a user is manipulating the virtual objectdisplayed in the user interface by contacting the touch-screen in thedirection shown by arrow 420 and arrow 422. By depressing the shiftbutton, the user may set the device to an active mode of interactionwith the graphical object. In the active mode of interaction, thegraphical object may become “grippable.” In this mode, the device mayrespond to finger gestures on the touch-screen with haptic feedbackrepresenting friction between the object and background surface.

In a passive mode of interaction, signals received from an accelerometermay not affect a gripped virtual object. However, in an active mode ofinteraction, accelerometer signals may still affect other virtualobjects in the user interface 410 (not shown in FIG. 4). Finger gesturesmay now activate a friction profile of the virtual object 412. Forexample, when the user depresses the shift button and makes a flickinggesture on the touch-screen, the processor may respond by moving thegraphical object within the user interface and generating a hapticeffect representing friction between the virtual environment and thevirtual object. This calculation is based in part on the frictionprofile of the virtual object and the virtual environment.

Although the messaging device illustrated in FIGS. 3 and 4 utilizes ashift button to send signals to the processor associated with theselection of a mode of interaction, other methods and/or devices may beused to accomplish this. As one example, tilting the messaging device ina forward direction may be associated with an active mode ofinteraction, whereas tilting the messaging device in a rear directionmay be associated with passive mode of interaction. As another example,double-clicking on the touch-screen may select the active mode ofinteraction, while single-clicking on the touch-screen may select thepassive mode of interaction.

FIG. 5 is an illustration of shifting haptic feedback function betweenpassive and active modes according to one embodiment of the presentinvention. In FIG. 5, a messaging device 502 comprises an accelerometer(not shown in FIG. 5) and a touch-screen 506. The messaging device 502generates a graphical user interface, such as a virtual environment 510shown on the touch screen 506.

As shown in FIG. 5, a user is rotating or moving the messaging device502 in a circular motion indicated by the arrow 520. After the sensordetects the circular motion, it sends a signal to the processor (notshown in FIG. 5). The processor receives the sensor's signal anddetermines that the circular motion is associated with a change in themode of interaction.

After moving the device in the predetermined manner, the user interfacemay shift from an active to a passive mode of interaction, or from apassive to an active mode of interaction. After the user interfaceshifts its mode of interaction, the user may interact with the userinterface according to the new mode of interaction.

FIG. 6 is an illustration of shifting haptic feedback function betweenpassive and active modes according to one embodiment of the presentinvention. In FIG. 6, a messaging device 602 comprises an accelerometer(not shown in FIG. 6) and a touch-screen 606. The messaging device 602generates a graphical user interface, such as a virtual environment 610shown on the touch screen 606.

As shown in FIG. 6, a user is jostling or shaking the messaging device602 in a circular motion indicated by the arrow 620. After the sensordetects the jostling motion, the sensor sends a signal to the processor(not shown in FIG. 6). The processor receives the sensor signal anddetermines that the jostling motion is associated with a change in themode of interaction.

After moving the device in the predetermined manner, the user interfacemay shift from a passive to an active mode of interaction, or from anactive to a passive mode of interaction. After the mode of interactionis switched, the user may interact with the graphical user interfaceaccording to the new mode of interaction. For example, in oneembodiment, the user may select a passive mode of interaction whiletyping an email. Then, wishing to send the email, the user may rotate orotherwise move the device in a predetermined manner, causing the deviceto switch to an active mode of interaction, in which the email may beshown on the display as a virtual ball. The user may then flick the ballto send the email; this action may trigger the processor to calculate ahaptic signal which generates a jolt, symbolizing the weight of thevirtual ball. Then, as the virtual ball travels off the display, thedevice sends the email. The user may then rotate the device or move thedevice in another predetermined manner to switch the device back topassive mode of interaction.

Embodiments of the present invention can be implemented in digitalelectronic circuitry or in computer hardware, firmware, software, or incombinations of these. In one embodiment, a computer may comprise aprocessor or processors. The processor comprises a computer-readablemedium, such as a random access memory (RAM) coupled to the processor.The processor executes computer-executable program instructions storedin memory, such as executing one or more computer programs formessaging. Such processors may comprise a microprocessor, a digitalsignal processor (DSP), an application-specific integrated circuit(ASIC), field programmable gate arrays (FPGAs), and state machines. Suchprocessors may further comprise programmable electronic devices such asPLCs, programmable interrupt controllers (PICs), programmable logicdevices (PLDs), programmable read-only memories (PROMs), electronicallyprogrammable read-only memories (EPROMs or EEPROMs), or other similardevices.

Such processors may comprise, or may be in communication with media,such as computer-readable media, that may store instructions that, whenexecuted by the processor, can cause the processor to perform the stepsdescribed herein as carried out, or assisted, by a processor.Embodiments of computer-readable media may comprise, but are not limitedto, an electronic, optical, magnetic, or other storage or transmissiondevice capable of providing a processor, such as the processor in a webserver, with computer-readable instructions. Other examples of mediacomprise, but are not limited to, a floppy disk, CD-ROM, magnetic disk,memory chip, ROM, RAM, ASIC, configured processor, all optical media,all magnetic tape or other magnetic media, or any other medium fromwhich a computer processor can read. Also, various other devices mayinclude computer-readable media, such as a router, private or publicnetwork, or other transmission device. Both the processor and theprocessing described may be in and/or dispersed through one or morestructures. The processor may comprise code for carrying out one or moreof the methods (or parts of methods) described herein.

General

The foregoing description of the embodiments, including preferredembodiments, of the invention has been presented only for the purpose ofillustration and description and is not intended to be exhaustive nor tolimit the invention to the precise forms disclosed. Numerousmodifications and adaptations thereof will be apparent to those skilledin the art without departing from the spirit and scope of the invention.

That which is claimed is:
 1. A system comprising: a display deviceconfigured to display a virtual object within a graphical environment; ahaptic output device; and a processor configured to: detect userinteraction with the virtual object; determine whether the userinteraction is detected while the system is in an active mode ofinteraction or a passive mode of interaction, wherein the active mode ofinteraction is a mode in which the system moves the virtual objectwithin the graphical environment based on the user interaction, andwherein the passive mode of interaction is a mode in which the userinteraction causes the system to simulate a surface feature of thevirtual object; in response to a determination that the user interactionis received while the system is in the active mode of interaction,determine a haptic effect based on at least one of a simulated frictionexperienced by the virtual object while it is moving in the graphicalenvironment or a simulated impact experienced by the virtual object; inresponse to a determination that the user interaction is received whilethe system is in the passive mode of interaction, determine the hapticeffect based on the simulated surface feature of the virtual object; andcontrol the haptic output device to output the haptic effect.
 2. Thesystem of claim 1, wherein the surface feature is a simulated texture ofthe virtual object.
 3. The system of claim 1, wherein the system is amobile device, the display device comprises a touchscreen, and the userinteraction is a touch input applied by a user on the touchscreen. 4.The system of claim 3, wherein when the mode of interaction comprises anactive mode of interaction the haptic effect is configured to simulatecollision between the user and the virtual object.
 5. The system ofclaim 3, wherein when the mode of interaction comprises a passive modeof interaction the haptic effect is configured to simulate a texture ofthe virtual object at a location of the touchscreen touched by the user.6. The system of claim 3, wherein the virtual object comprises data anda shape of the virtual object is determined based on a type of the data.7. The system of claim 3, further comprising a sensor configured todetect movement of the mobile device and wherein the processor isconfigured to determine a change in the mode of interaction based inpart on the movement.
 8. The system of claim 7, wherein the processor isconfigured to determine a haptic effect based in part on the change inmode of operation.
 9. The system of claim 1, wherein the graphicalenvironment comprises a plurality of virtual objects, and wherein eachof the plurality of virtual objects is associated with a type of data.10. A method comprising: displaying a virtual object within a graphicalenvironment; detecting user interaction with the virtual object;determining whether the user interaction is detected while a system isin an active mode of interaction or a passive mode of interaction,wherein the active mode of interaction is a mode in which the systemmoves the virtual object within the graphical environment based on theuser interaction, and wherein the passive mode of interaction is a modein which the user interaction causes the system to simulate a surfacefeature of the virtual object; in response to a determination that theuser interaction is received while the system is in the active mode ofinteraction, determine a haptic effect based on at least one of asimulated friction experienced by the virtual object while it is movingin the graphical environment or a simulated impact experienced by thevirtual object; in response to a determination that the user interactionis received while the system is in the passive mode of interaction,determine the haptic effect based on the simulated surface feature ofthe virtual object; and controlling a haptic output device to output thehaptic effect.
 11. The method of claim 10, wherein the surface featureis a simulated texture of the virtual object.
 12. The method of claim10, wherein the displaying is on a display of a mobile device, thedisplay comprising a touchscreen, and the user interaction is a touchinput applied by a user on the touchscreen.
 13. The method of claim 12,wherein when the mode of interaction comprises an active mode ofinteraction the haptic effect is configured to simulate collisionbetween the user and the virtual object.
 14. The method of claim 12,wherein when the mode of interaction comprises a passive mode ofinteraction the haptic effect is configured to simulate a texture of thevirtual object at a location of the touchscreen touched by the user. 15.The method of claim 12, wherein the virtual object comprises data and ashape of the virtual object is determined based on a type of the data.16. The method of claim 12, further comprising detecting a movement ofthe mobile device and determining a change in the mode of interactionbased in part on the movement.
 17. The method of claim 16, furthercomprising determining a haptic effect based in part on the change inmode of operation.
 18. The method of claim 10, wherein the graphicalenvironment comprises a plurality of virtual objects, and wherein eachof the plurality of virtual objects is associated with a type of data.19. A non-transitory computer readable medium comprising program code,which when executed by a processor is configured to cause the processorto: display a virtual object within a graphical environment; detect userinteraction with the virtual object; determine whether the userinteraction is detected while a system is in an active mode ofinteraction or a passive mode of interaction, wherein the active mode ofinteraction is a mode in which the system moves the virtual objectwithin the graphical environment based on the user interaction, andwherein the passive mode of interaction is a mode in which the userinteraction causes the system to simulate a surface feature of thevirtual object; in response to a determination that the user interactionis received while the system is in the active mode of interaction,determine a haptic effect based on at least one of a simulated frictionexperienced by the virtual object while it is moving in the graphicalenvironment or a simulated impact experienced by the virtual object; inresponse to a determination that the user interaction is received whilethe system is in the passive mode of interaction, determine the hapticeffect based on the simulated surface feature of the virtual object; andcontrol a haptic output device to output the haptic effect.